$I n t e r a c t i v e V i s u a l i z a t i o n o f W h y E i g e n v e c t o r s M a t t e r$

You've probably heard the word eigenvectors hundreds of times in class and have been asked to calculate it hundreds more. But why is this concept so central in Linear Algebra? Why is it that we study the eigenvector of a matrix so often?

In this post, we'll see how Eigenvectors help us immediately understand what a linear function will do to an input. We'll do so by playing with an interactive visualization that allows us to see just that.

Quick Refresher

Here, applying

A

on its eigenvector

v

leades to a new vector

λ v

is in the same direction as

v

. Image Source: Wikipedia.

Let's do a quick refresher to begin with. The eigenvector of a linear function

A

is just the vector

v

s.t.

A v = λ v

for some constant

λ

which we call the eigenvalue. At a high level, the eigenvector is just a dimension along which the linear function only stretches its input (for real valued eigenvalues).

This is great and all but what can we do once we find an Eigenvector? What does it tell us about the underlying Matrix?

Linear Functions Pull Inputs Towards the Dominant Eigenvector

Let's start with an example. Take the following linear function:

A = [\begin{matrix} 3 & 2 \\ 1 & 4 \end{matrix}]

Let's see what happens when we apply

A

repeatedly on the input

[\begin{matrix} 0 \\ 1 \end{matrix}]

{% comming soon!! %}

To play with this visualization, do the following:

Drag the slider to increase the number of times we apply $A$ .
Notice how the output vector tilts towards $v_{1}$ , an eigenvector of $A$ .

So just by knowing an eigenvector of

A

(namely the dominant eigenvector), we can get a sense of what

A

does -

A

pulls its input towards the axis of the dominant eigenvector.

How is this happening? And why is it only towards one eigenvector?

Breaking Up the Input With an Eigenbasis

You'll be surprised to see that this behavior comes very naturally from the properties of linear functions. Let's see this with an example.

Let's keep the same matrix/linear function

A = [\begin{matrix} 3 & 2 \\ 1 & 4 \end{matrix}]

, and analyze applying

A

three times on the input

v = [\begin{matrix} 0 \\ 1 \end{matrix}]

(i.e.

A^{3} v

In the standard way, we'd just use standard matrix multiplication to find

A^{3} v .

The standard way to do this is to simply follow the rules of multiplication and carry out

A (A (A (v))) .

But instead, let's do this a different way using eigenvectors.

In the following discussion we will split

v

into a linear combination of

A

's eigenvectors. We then apply

A^{3}

to each of these pieces and combine the result.

We know that that any vector

v

can be written as the sum of the eigenvectors of

A

. After all, eigenvectors are linearly independent and form a basis for the space (if the matrix

A

is diagonalizable, which it is). If

v_{1}

and

v_{2}

are the eigenvectors of

A,

we can break up

v

as:

v = c_{1} \cdot v_{1} + c_{2} \cdot v_{2}

for some constants

c_{1}

and

c_{2} .

We first split

v

into its eigenvector subcomponents.

When we have this representation, we can then rethink

A^{3} v

as:

A^{3} v = A^{3} (c_{1} \cdot v_{1} + c_{2} \cdot v_{2})

or more simply:

A^{3} v = c_{1} \cdot A^{3} v_{1} + c_{2} \cdot A^{3} v_{2}

We then carry out

A^{3} v_{1}

and

A^{3} 3 v_{2} .

We then carry out the computation of

A^{3} v_{1}

and

A^{3} v_{2}

. Thanks to

v_{1}

and

v_{2}

being eigenvectors, we have:

A^{3} v_{1} = λ_{1}^{3} v_{1}

A^{3} v_{2} = λ_{2}^{3} v_{2}

We finally combine the results to get

A^{3} v .

We then finally combine the results to get

A^{3} v .

We find:

A^{3} v = c_{1} λ_{1}^{3} v_{1} + c_{2} λ_{2}^{3} v_{2}

Dominant Eignevalues and Eigenvectors

Now, what happens when

| λ_{1} |

is larger than

| λ_{2} |

(i.e. there exists a dominant eigenvalue)? In this example,

λ_{1} = 5

and

λ_{2} = 2.

Let's now display what it would look like to carry out

A^{3} v

when we have this difference in eigenvalues.

The interaction below shows this setup:

{% include eigenvectors.html %}

Drag the slider to increase or decrease the number of times we apply $A$ on $v .$
Notice how "Output Eigenvector 1" and "Output Eigenvector 2" change at different rates.
Notice how "Final Output Vector" tilts towards "Output Eigenvector 1" as you drag the slider to the right.

We thus see that when there's one eigenvalue larger than the other (

| λ_{1} | > | λ_{2} |

), the linear function pushes its inputs towards the eigenvector associated with that large eigenvalue ("Output EigenVector One"). The more times we apply

A

, the larger this effect.

Note this "push" effect will only happen towards this eigenvector with the largest eignevalue - not any of the other eigenvectors.

Why this happens

This tilt towards "Output Vector One" happens due to exponential growth.

λ_{1}^{x}

grows much faster than

λ_{2}^{x}

. As such the more times we apply

A

(

x

in our exponentials), the bigger the difference between

λ_{1}^{x}

and

λ_{2}^{x} .

Hence the

v_{1}

term has much more weight in the final sum. This increasing difference is shown in the plot below.

Due to the power of exponentials, the dominant eigenvector will play a bigger and bigger role the more times we apply A. Notice how the distance between the two expontial functions increases with x.

Conclusion

So, just using the properties of linear functions, we are able to see why eigenvectors are so important. They show us where a linear function will "push" its inputs.

If you've enjoyed this post on eigenvectors, check out the following additional posts on the topic I've written:

Thanks for reading!

Caveat

Everything I've discussed is for real eigenvalues.
This only applies for matrices that are diagonalizable.

Credits

Thanks to Luis Serrano, Rouzbeh Shirvani, and Pranav Ramkrishnan for feedback.

BY Dhruv Parthasarathy(Director of AI Programs @ Udacity)